Methods, systems, and media for identifying abusive content items

ABSTRACT

Methods, systems, and media for identifying abusive content items are provided. In some embodiments, the method comprises: receiving, at a server from a user device associated with a user, a request to provide user-generated content on a media content platform; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criterion; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criterion, adding a review request to a queue of a reviewing user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision responsive to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device that indicates the at least one violated policy and a penalty to be administered to the user account corresponding to the user and (ii) a corresponding action is taken that inhibits the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device that indicates that the user-generated content is eligible for providing on the media content platform.

TECHNICAL FIELD

The disclosed subject matter relates to methods, systems, and media foridentifying abusive content items. More particularly, the disclosedsubject matter relates to determining whether user-generated contentshould be manually reviewed by one or more reviewers or whether manualreview of the user-generated content can be bypassed based on one ormore criterion and determining whether the user-generated content isabusive content such that a corresponding action can be taken on theuser-generated content.

BACKGROUND

Media content platforms have become the primary mechanism by which usersdiscover and share digital media content. For example, a content creatormay upload a video or other content item that the content creator hascreated to a media content platform, which can then make the uploadedcontent item available to viewers (e.g., by streaming the content item,by downloading the content item, etc.).

While many user-generated channels on a media content platform containoriginal, non-abusive content, there can be abusive content thatviolates a policy of the media content platform, such as misleading,repetitive, racy, pornographic, infringing, and/or “clickbait” content,and that targets unsuspecting users. For example, a particular contentitem may include objectionable content, such as violent content,objectionable language, adult content, drug-related content, gamblingcontent, etc. that the media content platform may want to remove orotherwise prevent its users from accessing. As another example, a mediacontent platform may want to block particular features of the servicefrom being used (e.g., features that allow a content creator to collectrevenue, and/or any other suitable features) with content items thatviolate these particular policies.

However, it can be difficult to identify content items that violatepolicies. For example, while a service may use human reviewers tomanually review uploaded content items, it can be time-intensive toreview each and every piece of uploaded content to identify policyviolations. In such cases, a content item may be available on a mediacontent platform for any number of days or weeks before a manual reviewprocess indicates that the content item should be blocked or removed dueto a policy violation of the media content platform.

Accordingly, it is desirable to provide new methods, systems, and mediafor identifying abusive content items.

SUMMARY

Methods, systems, and media for identifying abusive content items areprovided.

In accordance with some embodiments of the disclosed subject matter, amethod for identifying abusive content items is provided, the methodcomprising: receiving, from a user device associated with a user, arequest to provide user-generated content on a media content platform;in response to receiving the request, determining whether theuser-generated content is to be reviewed by one or more reviewing usersbased on one or more bypass criterion; in response to determining thatthe user-generated content is to be reviewed based on the one or morebypass criterion, adding a review request to a queue of a reviewinguser, wherein the review request includes the user-generated content andinformation associated with a user account corresponding to the user;and determining whether a decision responsive to the review requestindicates that the user-generated content is to be provided on the mediacontent platform, wherein: in response to determining that the decisionindicates that the user-generated content item violates at least onepolicy associated with the media content platform, (i) a first messageis transmitted to the user of the user device that indicates the atleast one violated policy and a penalty to be administered to the useraccount corresponding to the user and (ii) a corresponding action istaken that inhibits the user-generated content from being provided onthe media content platform; and, in response to determining that thedecision indicates that the user-generated content item does not violatethe at least one policy associated with the media content platform, asecond message is transmitted to the user of the user device thatindicates that the user-generated content is eligible for providing onthe media content platform.

In some embodiments, the user is a content creator and the request toprovide user-generated content includes an upload request to store theuser-generated content on a storage device associated with the mediacontent platform.

In some embodiments, the user is a content creator and the request toprovide user-generated content includes an initiation request to start alivestream using the media content platform.

In some embodiments, the user is a viewer and the request to provideuser-generated content includes a content feature that interacts with acontent creator.

In some embodiments, the method further comprises determining whetherthe user is eligible to provide the user-generated content on the mediacontent platform by: calculating a risk score that indicates a riskassociated with the user account of the user based on otheruser-generated content that has been previously provided to the mediacontent platform; and comparing the risk score with a threshold riskscore that corresponds to a content type of the of the user-generatedcontent, wherein the user is deemed eligible to provide theuser-generated content based on the comparison of the risk score withthe threshold risk score.

In some embodiments, the method further comprises determining whetherthe user is eligible to provide the user-generated content on the mediacontent platform based on audience information associated with the useraccount of the user.

In some embodiments, the method further comprises determining whetherthe user is eligible to provide the user-generated content on the mediacontent platform based on location information associated with the useraccount of the user.

In some embodiments, the method further comprises determining whetherthe user is eligible to provide the user-generated content on the mediacontent platform based on enforcement actions for previous policyviolations that are associated with the user account.

In some embodiments, the method further comprises determining whetherthe user is eligible to provide the user-generated content on the mediacontent platform by transmitting a second review request to the queue ofthe reviewing user, where the second review request includes informationassociated with the user account corresponding to the user and promptsthe reviewing user to determine whether the information associated withthe user account violates at least one policy associated with the mediacontent platform.

In some embodiments, the method further comprises, in response toreceiving the request to provide the user-generated content on the mediacontent platform, causing a timing indication to be presented thatindicates when the decision responsive to the review request is to beprovided.

In some embodiments, the method further comprises: receiving a flag froma viewer of the user-generated content on the media content platform,where the flag indicates a potential policy violation; and, in responseto receiving the subsequent review request, adding a subsequent reviewrequest to the queue of the reviewing user that previously reviewed theuser-generated content, where the subsequent review request includes theuser-generated content and the flag from the viewer and where theuser-generated content is restricted from being provided on the mediacontent platform until a decision responsive to the subsequent reviewrequest is received.

In some embodiments, determining whether the user-generated content isto be reviewed by one or more reviewing users based on one or morebypass criterion further comprises: identifying second user-generatedcontent that is similar to the user-generated content, wherein thesecond user-generated content has been previously reviewed; anddetermining that the user-generated content is not to be reviewed by theone or more reviewing users based on a similarity of the user-generatedcontent to the second user-generated content. In some embodiments, themethod further comprises: determining that the second user-generatedcontent was previously restricted from being made available on the mediacontent platform; and, in response to determining that the seconduser-generated content was previously restricted from being madeavailable on the media content platform, performing the correspondingaction that inhibits the user-generated content from being madeavailable on the media content platform, wherein the first messageindicates the restriction based on the second user-generated content.

In some embodiments, determining whether the user-generated content isto be reviewed by the one or more reviewing users further comprisesdetermining whether at least one of the user-generated content andcontent related to the user-generated content has been previouslyflagged by a threshold number of viewers.

In some embodiments, the method further comprises updating the useraccount associated with the user to indicate the at least one policyassociated with the media content platform violated by theuser-generated content.

In some embodiments, the penalty to be administered to the user accountcorresponding to the user is based on a number of previous policyviolations by the content creator.

In some embodiments, the penalty to be administered to the user accountcorresponding to the user based on a severity of the at least one policyassociated with the media content platform violated by theuser-generated content.

In some embodiments, the method further comprises: receiving, from theuser device, an appeal request of the decision, wherein the appealrequest includes appeal text; adding the appeal request that includesthe appeal text and the user-generated content to the queue of a secondreviewing user that is different from the reviewing user that providedthe decision indicating that the user-generated content is to berestricted from being provided on the media content platform; receiving,from a user device associated with the second reviewing user, an appealdecision corresponding to the appeal request; and determining at leastone corresponding action to be performed on the user-generated contentbased on the decision and the appeal decision.

In some embodiments, in response to determining that the decisionindicates that the user-generated content item does not violate the atleast one policy associated with the media content platform, the secondmessage indicates a time that the user-generated content isautomatically made available on the media content platform.

In accordance with some embodiments of the disclosed subject matter, asystem for identifying abusive content items is provided, the systemcomprising a memory and a hardware processor that, when executingcomputer executable instructions stored in the memory, is configured to:receive, from a user device associated with a user, a request to provideuser-generated content on a media content platform; in response toreceiving the request, determine whether the user-generated content isto be reviewed by one or more reviewing users based on one or morebypass criterion; in response to determining that the user-generatedcontent is to be reviewed based on the one or more bypass criterion, adda review request to a queue of a reviewing user, wherein the reviewrequest includes the user-generated content and information associatedwith a user account corresponding to the user; and determine whether adecision responsive to the review request indicates that theuser-generated content is to be provided on the media content platform,wherein: in response to determining that the decision indicates that theuser-generated content item violates at least one policy associated withthe media content platform, (i) a first message is transmitted to theuser of the user device that indicates the at least one violated policyand a penalty to be administered to the user account corresponding tothe user and (ii) a corresponding action is taken that inhibits theuser-generated content from being provided on the media contentplatform; and, in response to determining that the decision indicatesthat the user-generated content item does not violate the at least onepolicy associated with the media content platform, a second message istransmitted to the user of the user device that indicates that theuser-generated content is eligible for providing on the media contentplatform.

In accordance with some embodiments of the disclosed subject matter, anon-transitory computer-readable medium containing computer executableinstructions that, when executed by a processor, cause the processor toperform a method for identifying abusive content items, the methodcomprising: receiving, from a user device associated with a user, arequest to provide user-generated content on a media content platform;in response to receiving the request, determining whether theuser-generated content is to be reviewed by one or more reviewing usersbased on one or more bypass criterion; in response to determining thatthe user-generated content is to be reviewed based on the one or morebypass criterion, adding a review request to a queue of a reviewinguser, wherein the review request includes the user-generated content andinformation associated with a user account corresponding to the user;and determining whether a decision responsive to the review requestindicates that the user-generated content is to be provided on the mediacontent platform, wherein: in response to determining that the decisionindicates that the user-generated content item violates at least onepolicy associated with the media content platform, (i) a first messageis transmitted to the user of the user device that indicates the atleast one violated policy and a penalty to be administered to the useraccount corresponding to the user and (ii) a corresponding action istaken that inhibits the user-generated content from being provided onthe media content platform; and, in response to determining that thedecision indicates that the user-generated content item does not violatethe at least one policy associated with the media content platform, asecond message is transmitted to the user of the user device thatindicates that the user-generated content is eligible for providing onthe media content platform.

In accordance with some embodiments of the disclosed subject matter, asystem for identifying abusive content items is provided, the systemcomprising: means for receiving, from a user device associated with auser, a request to provide user-generated content on a media contentplatform; means for determining whether the user-generated content is tobe reviewed by one or more reviewing users based on one or more bypasscriterion in response to receiving the request; means for adding areview request to a queue of a reviewing user in response to determiningthat the user-generated content is to be reviewed based on the one ormore bypass criterion, wherein the review request includes theuser-generated content and information associated with a user accountcorresponding to the user; and means for determining whether a decisionresponsive to the review request indicates that the user-generatedcontent is to be provided on the media content platform, wherein: inresponse to determining that the decision indicates that theuser-generated content item violates at least one policy associated withthe media content platform, (i) a first message is transmitted to theuser of the user device that indicates the at least one violated policyand a penalty to be administered to the user account corresponding tothe user and (ii) a corresponding action is taken that inhibits theuser-generated content from being provided on the media contentplatform; and, in response to determining that the decision indicatesthat the user-generated content item does not violate the at least onepolicy associated with the media content platform, a second message istransmitted to the user of the user device that indicates that theuser-generated content is eligible for providing on the media contentplatform.

BRIEF DESCRIPTION OF THE DRAWINGS

Various objects, features, and advantages of the disclosed subjectmatter can be more fully appreciated with reference to the followingdetailed description of the disclosed subject matter when considered inconnection with the following drawings, in which like reference numeralsidentify like elements.

FIG. 1 shows an illustrative example of an information flow diagram foridentifying abusive content items in accordance with some embodiments ofthe disclosed subject matter.

FIG. 2 shows an illustrative example of a process for identifyingabusive content items in accordance with some embodiments of thedisclosed subject matter.

FIG. 3 shows a schematic diagram of an illustrative system suitable forimplementation of mechanisms described herein for identifying abusivecontent items in accordance with some embodiments of the disclosedsubject matter.

FIG. 4 shows a detailed example of hardware that can be used in a serverand/or a user device of FIG. 3 in accordance with some embodiments ofthe disclosed subject matter.

DETAILED DESCRIPTION

In accordance with various embodiments, mechanisms (which can includemethods, systems, and media) for identifying abusive content items areprovided. More particularly, the disclosed subject matter relates todetermining whether user-generated content should be manually reviewedby one or more reviewers or whether manual review of the user-generatedcontent can be bypassed based on one or more criterion and determiningwhether the user-generated content is abusive content such that acorresponding action can be taken on the user-generated content.

Prior abusive content detection systems typically allow user-generatedcontent to be published or otherwise made available prior to suchcontent being reviewed, which introduces a great amount of risk. Themechanisms disclosed herein can provide a multi-stage abuse mitigationapproach that inhibits user-generated content from being made availableon a media content platform, for example, without requiring human reviewof each piece of content while ensuring that user-generated content thatdoes not meet one or more bypass criterion is manually reviewed.

In some embodiments, the mechanisms described herein can receive acontent item that is a user-generated content item from a creator of thecontent item. The user-generated content item can include, for example,any suitable type of user-generated content, such as a video contentitem or an audio content item generated by a content creator, a commententered by a viewer of media content in connection with an online chator a comments section, use of a content feature to generateuser-generated content, and/or any other suitable type of user-generatedcontent. It should be noted that user-generated content can include anysuitable content, such as video content, audio content, textual content,live streamed video content, etc.

In some embodiments, the mechanisms can determine whether the contentitem is to be manually reviewed by one or more reviewers. In someembodiments, in response to determining that the user-generated contentis to be manually reviewed, the mechanisms can add the user-generatedcontent to a queue of content items to be reviewed by a reviewer (e.g.,a reviewing user at a suitable computing device). In some embodiments, areviewer can then receive the user-generated content from the queue ofcontent items and can review the user-generated content to determine ifthe user-generated content violates a policy of the media contentplatform. For example, in some embodiments, the reviewer can determinewhether the user-generated content includes any particular types ofobjectionable content (e.g., violent content, objectionable language,hate speech, adult content, and/or any other suitable type ofobjectionable content). In some embodiments, the reviewer can thentransmit a decision indicating whether the user-generated contentviolates any policies of the media content platform. In someembodiments, the mechanisms can then determine whether the content itemis to be posted to the media content platform in connection with thefeature.

It should be noted that, in some embodiments, the mechanisms can, inresponse to receiving a decision from a first reviewer indicating thatthe user-generated content violates a policy of the media contentplatform, add the user-generated content to another queue of contentitems to be reviewed by a different reviewer (e.g., a verificationreviewer, a reviewer having more experience than the initial reviewer,etc.). In continuing this example, a corresponding action, such asremoving the user-generated content from the media content platform orotherwise inhibiting the user-generated content from being madeavailable on the media content platform, can be taken in response toreceiving decisions from the first reviewer and the second, differentreviewer both indicating that the user-generated content violates apolicy of the media content platform.

In some embodiments, the mechanisms can determine whether the contentitem is to be manually reviewed based on any suitable information. Forexample, in some embodiments, the mechanisms can determine whether thecontent item is to be manually reviewed based on information associatedwith the content creator, such as whether the content creator haspreviously uploaded content items that have been approved, whether thecontent creator has previously uploaded content items that have violatedpolicies of the media content platform, and/or any other suitableinformation associated with the content creator. As another example, insome embodiments, the mechanisms can determine whether the content itemis to be manually reviewed based on a similarity of the content item topreviously reviewed content items. As a more particular example, in someembodiments, the mechanisms can determine that the content item issimilar to a previously rejected content item and can determine that theuploaded content item is also to be rejected. In some embodiments, themechanisms can determine criteria used to determine if content items areto be added to a queue of items to be manually reviewed based on anumber of received content items, thereby allowing an amount of manualreview that is assigned to be adjusted.

In some embodiments, in an instance in which a review of a content itemindicates that the content item violates a policy of the media contentplatform, the mechanisms can cause a message to be transmitted to thecontent creator that indicates any suitable information. For example, insome embodiments, the message can indicate a policy that was violated,timestamp information indicating a portion of the content item thatviolates the policy, a penalty for the policy violation, and/or anyother suitable information. Note that, in some embodiments, themechanisms can determine the penalty based on any suitable information,such as a number of previous violations by the content creator, aseverity of the violation, and/or any other suitable information.Additionally, in some embodiments, the mechanisms can cause the contentitem to be blocked from the media content platform and/or removed fromthe media content platform.

In some embodiments, in response to determining that the user-generatedcontent or the use of a feature for generating user-generated content isallowable (e.g., in response to a reviewer determining that the contentdoes not violate a policy of the media content platform, in response toa determination that the content is likely to not violate a policy ofthe media content platform, etc.), the mechanisms can transmit aspecific time in which the user-generated content is posted or otherwisemade available on the media content platform (e.g., rather thanpublishing or posting the user-generated prior to any review forviolation of one or more policies of the media content platform, ratherthan a delay after publishing or posting the content in which a reviewof the content is submitted, etc.). Alternatively, the mechanisms cantransmit a specific time in which the user-generated content can beposted or published at the direction of the content creator.

In some embodiments, the mechanisms can receive an appeal of a decisionfrom a content creator. In some such embodiments, the mechanisms can addthe appeal to a queue of appeals, and the appeal can be assigned to areviewer. In some embodiments, any suitable information can be providedto a reviewer of the appeal, such as the content item, informationassociated with the original decision that is being appealed (e.g.,information indicating the policy that was violated, timing informationindicating portions of the content item that violated the policy, and/orany other suitable information), text from the content creator (e.g., anargument from the content creator, and/or any other suitable text),and/or any other suitable information. In some embodiments, a decisionrelated to the appeal can then cause the original decision to be upheldor, conversely, can cause the original decision to be overturned. Ininstances in which the original decision is overturned, and in which thecontent item had previously been blocked, the mechanisms can cause thecontent item to be made available on the media content platform.

Note that, in some embodiments, a content item can be received from acontent creator in which the content item is to be provided inconnection with use of a feature by the content creator. For example, insome embodiments, the content item can be a media content item receivedby a media content platform for providing to viewers in connection witha particular feature. As a more particular example, in some embodiments,the feature can include providing a membership service to viewers of achannel of content associated with the media content platform, wheremembership can allow members to access any suitable perks associatedwith memberships (e.g., access to special content, access to a chat roomor forum with other members, access to custom emojis or badges, and/orany other suitable perks). As another more particular example, in someembodiments, the feature can include allowing a content creator toprovide merchandise (e.g., branded apparel, and/or any other suitablebranded merchandise) associated with the content item and/or associatedwith a channel of content to which the content item belongs. Additionaldetails regarding features are described below in more detail inconnection with FIG. 2. In some such embodiments, the mechanisms candetermine whether the content item is to be manually reviewed todetermine whether the content item is eligible to be provided to usersin connection with use of the feature. For example, in some embodiments,the mechanisms can determine whether the content creator is eligible touse the feature, determine whether the content item is to be manuallyreviewed, determine whether the content item is to be blocked from beingprovided in connection with the feature, etc., as described below inmore detail in connection with FIG. 2.

In a more particular example, prior to making user-generated contentavailable on a media content platform, the mechanisms can use humanreviewers and machine reviewers to determine whether a user account iseligible to access media content platform features to generateuser-generated content. This can, for example, restrict access to mediacontent platform features to user accounts that have been evaluated asbeing clean or non-abusive user accounts.

Note that, in some embodiments, content items as referred to herein canrefer to any suitable type of user-generated content, such as mediacontent generated and uploaded by a content creator, an offer of amembership perk offered by a content creator in exchange for purchase ofa membership associated with a channel of content produced by thecontent creator, comments entered by viewers of content on a commentssection or social networking post, and/or any other suitable type ofcontent. That is, in some embodiments, the mechanisms described hereincan be used to determine whether any suitable type of user-generatedcontent is to be manually reviewed, for example, prior to posting theuser-generated content in a particular page, forum, comments section,etc. For example, the mechanisms described herein can be used to reviewcomments (e.g., comments submitted on a comment section associated witha content item, comments submitted in a chat forum, comments submittedon a social networking post, and/or any other suitable comments). As amore particular example, in some embodiments, the mechanisms describedherein can be used to determine whether a particular comment is to beadded to a queue of comments to be manually reviewed or to determinewhether a comment is to be deleted. As another example, in someembodiments, the mechanisms described herein can be used to review perksoffered by a content creator in connection with membership to a channelof content provided by the content creator. As a more particularexample, in some embodiments, the mechanisms described herein can beused to determine whether a particular perk offered by a content creatoris to be added to a queue of perks to be manually reviewed and/or todetermine whether the particular perk is to be cancelled (e.g., due toviolating any particular policy associated with the media contentplatform).

Turning to FIG. 1, an illustrative example 100 of an information flowdiagram for identifying abusive content items is shown in accordancewith some embodiments of the disclosed subject matter. As illustrated,in some embodiments, blocks of process 100 can be perform on a contentcreator device 308, a server 302, and/or a reviewer device 310, as shownin and described below in connection with FIG. 3.

At 102, content creator device 308 can transmit a user-generated contentitem to server 302. In some embodiments, the user-generated content itemcan include any suitable type of content, such as a video, a playlist ofvideos, a link to a live stream of a video, an image, an animation, aphoto, a slideshow of photos, a document, and/or any other suitable typeof content. In a more particular example, content creator device 308 canupload the user-generated content item for publishing with a channelassociated with a user account of a video sharing service. As anotherexample, in some embodiments, the user-generated content item caninclude a comment submitted by a user of content creator device 308 tobe posted in a chat room, a live chat, in a comments section associatedwith a particular page, media content item, or social networking post,and/or any other suitable type of comment. In some embodiments, contentcreator device 308 can transmit the user-generated content item inconnection with a user account associated with content creator device308 (e.g., a user account on a media content platform associated withserver 302, and/or any other suitable user account). Note that, in someembodiments, server 302 can be associated with a media content platformthat hosts uploaded content and that streams or otherwise providescontent hosted by the media content platform to one or more userdevices.

It should be noted that, in some embodiments, in response to receivingthe user-generated content item, server 302 can determine whether theuser-generated content item is to be reviewed by one or more reviewingusers. In turn, server 302 can cause a notification or other suitableinterface to be presented on content creator device 308 that indicatesthe user-generated content item or access to a feature for generatinguser-generated content is being reviewed. For example, the notificationcan indicate a particular amount of time until a decision from areviewer is received. In another example, the notification can indicatethat an updated notification will be presented upon approval of theuser-generated content, thereby providing the content creator at contentcreator device 308 with the control of publishing or otherwise makingthe user-generated content available on the media content platform.

At 104, server 302 can add the content item to a queue of content itemsto be reviewed. In some embodiments, the queue of content items cancorrespond to a queue of content items that are to be manually reviewedby human reviewers, and/or reviewed in any other suitable manner. Insome embodiments, server 302 can maintain the queue in any suitablemanner. For example, in some embodiments, content items can be added tothe queue based on a time at which the content item is received byserver 302 (e.g., such that items that are received later are placed ata later point in the queue, and/or in any other manner). As anotherexample, in some embodiments, content items can be added to the queuebased on information associated with a content creator associated withcontent creator device 308. As a more particular example, in someembodiments, server 302 can determine a popularity of the contentcreator (e.g., based on a number of subscribers to a channel of contentassociated with the content creator, based on a number of views ofcontent generated by the content creator, and/or based on any othersuitable information) and can place the content item in the queue basedon the popularity of the content creator (e.g., a content item receivedfrom a relatively more popular content creator can be inserted at arelatively higher position within the queue, and/or in any othersuitable manner). In some embodiments, server 302 can maintain the queueof content items based on any suitable combination of factors.

Note that, as described below in more detail in connection with block208 of FIG. 2, in some embodiments, server 302 can determine whether thecontent item is to be added to the queue. In some embodiments, server302 can add the content item to the queue of content items to bereviewed in response to determining that the content item is to bereviewed. Alternatively, in some embodiments, server 302 can determinethat the user-generated content item meets one or more bypass criterionsuch that the user-generated content item is not to be reviewed and isnot to be added to a queue of content items to be reviewed at 104. Insome such instances, server 302 can determine whether the user-generatedcontent item is to be posted or removed (or otherwise inhibited frombeing made available on the media content platform) using any suitabletechnique or combination of techniques, as described below in moredetail in connection with blocks 208 and 210 of FIG. 2. For example, insome embodiments, server 302 can determine that the user-generatedcontent item is substantially similar (e.g., a threshold amount ofsimilarity) to previously approved user-generated content and, inresponse, can allow the user-generated content to be made available onthe media content platform. In another example, in some embodiments,server 302 can determine that the user-generated content issubstantially similar (e.g., a threshold amount of similarity) topreviously removed user-generated content and, in response, canautomatically flag, remove, or otherwise inhibit the user-generatedcontent from being provided on the media content platform withouttransmitting the user-generated content to one or more reviewers.

It should be noted that, in some embodiments, server 302 can receive anindication, such as a flag from any suitable user of the media contentplatform, that user-generated content may violate a policy of the mediacontent platform (e.g., for containing objectionable content within theuser-generated content). In response to receiving such an indication,server 302 can add the user-generated content to the queue of contentitems to be reviewed by one or more reviewers (e.g., a reviewing user ata suitable computing device). This can, for example, ensure thatuser-generated content is reviewed by one or more reviewers.

At 106, reviewer device 310 can receive an indication of the contentitem from the queue. In some embodiments, reviewer device 310 canreceive the indication in any suitable manner. For example, in someembodiments, reviewer device 310 can receive the indication via a userinterface through which a user of reviewer device 310 views a contentitem and transmits a decision related to the content item (e.g., adecision indicating that the content item violates a particular policyassociated with a media content platform, a decision indicating that thecontent item does not violate any policies associated with a mediacontent platform, and/or any other suitable decision). As a moreparticular example, in some embodiments, reviewer device 310 can receivea notification that indicates that the content item has reached the topof the queue of items to be reviewed and/or that the content item isbeing assigned to a user of reviewer device 310 for review. In some suchembodiments, the notification can include any suitable information, suchas a link to the content item, a name of a content creator associatedwith content creator device 308, a topic associated with the contentitem, information associated with the content creator (e.g., a number ofpreviously uploaded and/or approved content items created by the contentcreator, a number of times content previously uploaded by the contentcreator has been associated with a policy violation, and/or any othersuitable information), and/or any other suitable information.

Note that, in some embodiments, a user of reviewer device 310 can selectthe content item based on the indication in any suitable manner (e.g.,by selecting a link presented in the indication, and/or in any othersuitable manner), which can cause the content item to be presented in avideo player window or any other suitable user interface on reviewerdevice 310.

At 108, reviewer device 310 can transmit a decision relating to thecontent item. For example, in some embodiments, the decision canindicate any suitable information relating to the content item, such aswhether the content item includes content that violates one or morepolicies associated with a media content platform to which the contentitem was uploaded. As a more particular example, in some embodiments,the policies can include any suitable policies, such as the content itemincluding violent content, sexual content, abusive language, hatespeech, and/or any other suitable type of objectionable content.

In some embodiments, the transmitted decision can include any suitableinformation, such as an indication of a type of objectionable contentincluded in the content item, an indication of one or more portions ofthe content item that include the objectionable content (e.g., based ontimestamps of a video, frames of a video, and/or in any other suitablemanner), and/or any other suitable information. Note that, in someembodiments, the decision can indicate that the content item does notviolate any policies.

At 110, server 302 can update an account associated with the contentcreator device based on the decision received from reviewer device 310.For example, in an instance in which the decision indicates that thecontent item violates a particular policy, server 302 can update a tallythat indicates a number of times the content creator has uploadedcontent that violates a policy associated with the media contentplatform. As another example, in an instance in which the decisionindicates that the content item does not violate a particular policy,server 302 can update a tally that indicates a number of times thecontent creator has uploaded content that does not violate any policiesassociated with the media content platform (that is, a number of timesthe content creator has uploaded content that has been approved by areviewer).

In some embodiments, the decision can then be viewed by a user ofcontent creator device 308. For example, in some embodiments, the userof content creator device 308 can log into a user account associatedwith content creator device 308 and the media content platform and canview the decision in a user interface associated with the user account.In another example, in response to the decision indicating that theuser-generated content or the use of a feature for generatinguser-generated content is allowable (e.g., in response to a reviewerdetermining that the content does not violate a policy of the mediacontent platform, in response to a determination that the content islikely to not violate a policy of the media content platform, etc.), thedecision can be presented on content creator device 308 along with aspecific time in which the user-generated content is posted or otherwisemade available on the media content platform (e.g., rather thanpublishing or posting the user-generated prior to any review forviolation of one or more policies of the media content platform, ratherthan a delay after publishing or posting the content in which a reviewof the content is submitted, etc.). Alternatively, in some embodiments,the decision can be presented along with a specific time in which theuser-generated content can be posted or published at the direction ofthe content creator.

It should be noted that, in some embodiments, in response to thereviewer providing a decision in which the user-generated content isdeemed to violate a policy of the media content platform, server 302 canadd the content to a queue of content items to be reviewed by anotherreviewer. It should also be noted that, in some embodiments, theadditional reviewer can be a different reviewer having more experiencein comparison with the reviewer that provided the initial decision. Insuch an embodiment, server 302 can remove or otherwise inhibit theuser-generated content from being provided by the media content platformin response to receive two negative decisions indicating that theuser-generated content violates a policy of the media content platformfrom two different reviewers.

In some embodiments, the decision can be appealed by a user of contentcreator device 308. For example, in some embodiments, the decision canbe appealed in an instance in which the user of content creator device308 disagrees with the decision (e.g., in an instance in which thedecision indicates that the content item includes a particular type ofobjectionable content and/or violates a particular policy and in whichthe user disagrees, and/or in any other suitable instance). In someembodiments, blocks 112-118 as shown in FIG. 1 and described below canbe executed in an instance in which the decision is appealed. In someembodiments, blocks 112-118 can be omitted in an instance in which thedecision is not appealed.

At 112, content creator device 308 can transmit an indication that thedecision is being appealed. In some embodiments, content creator device308 can transmit the indication in any suitable manner. For example, insome embodiments, a user of content creator device 308 can select anysuitable button or user interface control associated with the decisionthat, when selected, cause a user interface for entering any suitabletext associated with an appeal of the decision (e.g., any suitablereasons contesting the decision, and/or any other suitable text). Insome such embodiments, the text associated with the appeal of thedecision can be transmitted in connection with the indication that thedecision is being appealed to server 302.

At 114, server 302 can add the content item to an appeals queue. In someembodiments, the content item can be added to the appeals queue in anysuitable manner. For example, in some embodiments, the content item canbe added to the appeals queue based on a time at which the indicationthat the decision is being appealed was transmitted by content creatordevice 308 at 112. As another example, in some embodiments, the contentitem can be added to the appeals queue based a relative popularity ofthe content creator associated with the content item, as described abovein connection with block 104.

At 116, reviewer device 310 can receive an indication of the contentitem from the appeals queue. In some embodiments, reviewer device 110can receive the indication of the content item in any suitable manner.For example, in some embodiments, reviewer device 110 can present anotification that indicates that a decision associated with the contentitem is being appealed by a content creator associated with the contentitem.

In some embodiments, a user of reviewer device 310 can select theindication in any suitable manner, and, in some embodiments, selectionof the indication can cause the content item to be presented on reviewerdevice 310 in connection with any suitable information associated withthe decision and/or any suitable information associated with the appeal.For example, in some embodiments, the information can include textassociated with the appeal (e.g., as described above in connection withblock 110), information associated with the decision that is beingappealed (e.g., information indicating the objectionable content or thepolicy violation indicated in the decision, information indicatingtimestamps at which particular objectionable content is indicated aspresented in the decision, and/or any other suitable information),and/or any other suitable information).

Note that, although FIG. 1 shows the same device (that is, reviewerdevice 310) as both reviewing the content item at 106 and reviewing anappeal of the content item at 116, in some embodiments, two differentreviewer devices can be used. That is, in some embodiments, a differentreviewer can review an appeal of a decision than a reviewer thattransmitted the decision that is being appealed. For example, adifferent reviewer at a different reviewer device can be used to reviewthe user-generated content in the appeals queue, where the differentreviewer is designated as having more experience than the initialreviewer that provided the negative decision and where an action, suchas suspending a user account or inhibiting the presentation of theuser-generated content, can be taken in response to both the initialreviewer and the different appeals reviewer providing negativedecisions.

At 118, reviewer device 310 can transmit a decision related to appeal ofthe content item. In some embodiments, the decision can indicate anysuitable information, such as that a user of reviewer device 310 agreeswith the decision being appealed, that a user of reviewer device 310disagrees with the decision being appealed, and/or any other suitableinformation.

At 120, server 302 can update the account associated with the contentcreator based on the decision related to the appeal. For example, in aninstance in which the decision being appealed indicated that the contentitem violates a particular policy, and in which the decision related tothe appeal disagrees with the decision being appealed, server 302 canupdate the account to reflect the disagreement between the twodecisions. As a more particular example, in some embodiments, server 302can decrement a tally of a number of times the content creator hasuploaded content items that violate policies associated with the mediacontent platform. As another example, in an instance in which thedecision related to the appeal confirms the decision being appealed thatthe content item violates a particular policy, server 302 can performany suitable action(s), such as blocking the content item, prohibitingcontent creator device 308 from uploading content items for apredetermined duration of time (e.g., for a month, for ninety days,and/or any other suitable duration of time), and/or any other suitableactions. Additional details on actions that can be performed by server302 are described below in connection with block 224 of FIG. 2.

Turning to FIG. 2, an illustrative example 200 of a process foridentifying abusive content is shown in accordance with some embodimentsof the disclosed subject matter. In some embodiments, blocks of process200 can be executed on a server, such as a server associated with amedia content platform that hosts content items uploaded by one or morecontent creators.

Process 200 can begin at 202 by receiving, from a content creator, acontent item to be uploaded or posted in connection with a media contentplatform. For example, in some embodiments, the content item can be avideo content item or an audio content item created by the contentcreator that is to be uploaded the media content platform and madeavailable for viewing by other users of the media content platform. Asanother example, in some embodiments, the content item can be a commentsubmitted by a viewer of a content item hosted by the media contentplatform. As a more particular example, in some embodiments, the commentcan be a comment submitted by a viewer in connection with a chat room orlive chat associated with a particular channel of content, a commentsubmitted by a viewer to be posted in a particular comments section(e.g., a comments section associated with a particular media contentitem hosted by the media content platform, and/or any other suitablecomments section), and/or any other suitable comment. As yet anotherexample, in some embodiments, the content item can be a perk that isbeing offered via the media content platform by a content creator inconnection with a channel of content created by the content creator. Asa more particular example, the perk can be an offer of a particular typeof content (e.g., exclusive video content, behind-the-scenes content,apparel or other goods, and/or any other suitable type of content) thatwill be provided by the content creator in exchange for users purchasingmemberships or subscriptions to the channel of content.

In some embodiments, the uploaded content item can be uploaded inconnection with a particular feature of the media content platform, forexample, a feature the content creator is interested in using togenerate user-generated content. In some embodiments, the feature caninclude features associated with comments or messages that can betransmitted by viewers of the content item. For example, in someembodiments, the feature can be related to comments in a comment sectionassociated with the comment item and/or comments posted in a live chatassociated with the content item. As a more particular example, thefeature can include allowing a viewer of the content item to purchase a“super chat,” which can allow the viewer to pin a comment associatedwith the purchased “super chat” to a particular location in a chat orcomments section (e.g., at the top of a live chat, in a top position ofa comments section, and/or in any other suitable location). Continuingwith this example, in some embodiments, the feature can includedetermining a duration of time a “super chat” comment is to be pinned tothe particular location based on an amount of payment paid by the viewerwho purchased the “super chat.” As another more particular example, insome embodiments, the feature can include allowing a viewer of thecontent item to purchase a “super sticker” that allows the viewer toinclude an animated image or icon in a comments section or live chatassociated with the content item. Continuing with this example, in someembodiments, the feature can include pinning particular “super sticker”images or icons at a particular location of a live chat or commentssection in response to determining that any particular criteria havebeen met (e.g., that the viewer paid more than a predetermined amountfor the “super sticker,” and/or any other suitable criteria). Note that,in some embodiments, animated images or icons that are available forselection to a viewer of a content item can be curated in any suitablemanner, for example, generated and made available by the contentcreator, by the media content platform, and/or by any other suitableentity. As yet another more particular example, in some embodiments, thefeature can include allowing a viewer of the content item to purchase a“super thanks.” In some such embodiments, purchase of a “super thanks”by a viewer can cause a selected animated image or icon (e.g., selectedby the viewer of the content item in connection with purchase of the“super thanks”) to be presented in connection with the content itemand/or can cause any suitable proportion of a payment by the viewer tobe shared with the content creator.

Note that, in some embodiments, the feature can allow the contentcreator to receive any suitable proportion of revenue from a purchased“super chat,” “super sticker,” and/or “super thanks.” For example, insome embodiments, the content creator can receive a proportion of therevenue from purchased options, and the media content platform canreceive the remainder of the revenue. Additionally or alternatively, insome embodiments, the feature can allow the content creator to specifyany suitable charitable organization to receive a proportion of therevenue.

In some embodiments, the feature can allow the content creator to offera membership service to viewers of content created by the contentcreator. For example, in some embodiments, the membership service canallow viewers who purchase a membership to receive any suitable perks inconnection with content created by the content creator. In someembodiments, the perks can include any suitable special offers availableto viewers who purchase memberships, such as access to special orexclusive content created by the content creator (e.g., only availableto viewers who purchase memberships), access to exclusive emojis orother animated icons that can be included in chat messages or commentssections associated with content created by the content creator, accessto exclusive chat forums where viewers who have purchased membershipscan chat with other viewers who have also purchased memberships, and/orany other suitable special offers available to members.

In some embodiments, the feature can allow the content creator to offermerchandise through the media content platform. For example, in someembodiments, the feature can allow the content creator to include linksto merchandise (e.g., apparel, mugs or cups, artwork, and/or any othersuitable merchandise) in a page which presents the content item or othercontent provided by the content creator through the media contentplatform. As a more particular example, the feature can allow thecontent creator to include links to one or more goods which areavailable for purchase in a video watch page which presents the contentitem. As another more particular example, the feature can allow thecontent creator to include links to one or more goods which areavailable for purchase in a page associated with a channel of contentassociated with the content creator.

In some embodiments, the feature can allow the content creator to offerone or more links to purchase tickets for a live show associated withthe content creator. For example, in an instance in which the contentitem is a music video associated with a particular band or other musicalgroup, the feature can allow links to purchase tickets for a concertassociated with the band or musical group to be presented in a page inwhich the content item is presented (e.g., below a video player windowin which the content item is presented, and/or in any other suitablelocation). Note that, in some embodiments, the feature can allow linksto purchase tickets to be presented in connection with any othersuitable page or content, such as a page associated with a channel ofcontent created by the content creator, and/or any other suitablecontent.

At 204, process 200 can determine whether the content creator iseligible to use the feature for generating user-generated content. Insome embodiments, process 200 can determine whether the content creatoris eligible to use the feature based on any suitable information andusing any suitable technique(s). For example, in some embodiments,process 200 can determine whether the content creator is eligible to usethe feature based on content items the content creator has previouslyuploaded to the media content platform. As a more particular example, insome embodiments, process 200 can use any suitable classifier or machinelearning technique to determine a risk score that indicates a likelihoodthat the content creator is likely to upload video content that includesone or more types of objectionable content based on previously uploadedcontent items. As a specific example, process 200 can determine that acontent creator with a risk score below a predetermined threshold is noteligible to use the feature. As another specific example, in someembodiments, process 200 can determine that a content creator with arisk score above a predetermined threshold is eligible to use thefeature. As another more particular example, in some embodiments,process 200 can determine a number of times the content creator haspreviously violated one or more policies associated with the mediacontent platform (e.g., policies that prohibit particular types ofobjectionable content, and/or any other suitable policies). As aspecific example, in some embodiments, process 200 can determine that acontent creator who has violated one or more policies of the mediacontent platform more than a predetermined threshold number of times(e.g., more than once, more than twice, more than once in the pastmonth, and/or any other suitable number of times) is not eligible to usethe feature.

As another example, in some embodiments, process 200 can determinewhether the content creator is eligible to use the feature based oninformation associated with the content creator, such as a size of theaudience of the content creator and/or a location associated with thecontent creator. As a more particular example, in some embodiments,process 200 can determine that content creators with a relatively smallaudience are not eligible to use the feature. As another more particularexample, in some embodiments, process 200 can determine that contentcreators with a relatively large audience are eligible to use thefeature. In some embodiments, a size of an audience can be determinedbased on any suitable information, such as a total number of views ofcontent items previously uploaded by the content creator, an averagenumber of views of content items previously uploaded by the contentcreator, a number of subscribers to a channel of content created by thecontent creator, and/or any other suitable metric.

As yet another example, in some embodiments, process 200 can determinewhether the content creator is eligible to use the feature based on amanual review of content posted by the content creator. As a moreparticular example, in some embodiments, a human reviewer can review oneor more content items previously posted by the content creator and/orthe content item uploaded by the content creator as described above inconnection with block 202 and can determine whether the content item(s)violate one or more policies of the media content platform. In someembodiments, in response to determining that the content item(s) violateone or more policies of the media content platform, process 200 candetermine that the content creator is not eligible to use the feature.

As still another example, in some embodiments, process 200 can determinewhether the content creator is eligible to use the feature based on anyother suitable information associated with the content creator and/orassociated with content items associated with the content creator. As amore particular example, in some embodiments, process 200 can determinewhether the content creator has been previously approved to join aprogram associated with the media content platform that allows thecontent creator to use particular features associated with membership inthe program. As another more particular example, in some embodiments,process 200 can determine whether an age of the content creator isgreater than a predetermined age (e.g., more than 18, and/or more thanany other suitable age). As yet another more particular example, process200 can determine whether content items associated with the contentcreator are associated with particular genres (e.g., content that isage-restricted, content that is targeted at children, music videocontent, and/or any other suitable genres of content). As a specificexample, in some embodiments, process 200 can determine that the contentcreator is not eligible to use the feature if content items associatedwith the content item typically belong to particular genres, such asage-restricted content, content targeted at children, music videocontent, etc.

As still another example, in some embodiments, process 200 can determinewhether the content creator is eligible to use the feature based on adetermination of whether viewers of content created by the contentcreator are satisfied or dissatisfied by use of the feature by thecontent creator. As a more particular example, in some embodiments, inan instance in which the feature includes providing perks to viewers whopurchase memberships associated with content produced by the contentcreator, process 200 can determine whether users who have purchasedmembership are generally satisfied or dissatisfied with their purchase.As another more particular example, in some embodiments, in an instancein which the feature includes allowing viewers to purchase a pinnedcomment, a pinned animated image, or other message or chat content,process 200 can determine whether users who have purchased pinnedmessage or chat content are generally satisfied or dissatisfied withtheir purchase. In some embodiments, process 200 can determine viewersatisfaction metrics in any suitable manner. For example, in someembodiments, process 200 can determine a number of viewers or aproportion of viewers who have cancelled a membership, a number ofviewers or a proportion of viewers who have left a negative review of anoffered perk, a number of viewers or a proportion of viewers who haveunsubscribed from a channel of content produced by the content creator,and/or any other suitable satisfaction metric. In some embodiments,process 200 can determine that the content creator is not eligible touse the feature in response to determining that a calculatedsatisfaction metric is below a predetermined threshold. As anotherexample, in some embodiments, process 200 can determine that the contentcreator is not eligible to use the feature in response to determiningthat more than a predetermined proportion of viewers (e.g., more than10%, more than 20%, and/or any other suitable proportion) have canceleda purchased membership or left a negative review in connection with anoffered perk or purchase option.

Note that, in an instance in which the content item corresponds to asubmitted comment, process 200 can determine whether a user whosubmitted the comment is eligible to post a comment to a particular chator particular comments section. For example, in an instance in which thecomment is submitted in connection with a chat or comments section thatrequires a subscription to a particular channel of content of the mediacontent platform or that requires any other suitable criteria be met,process 200 can determine whether the user who submitted the comment isa subscriber to the channel of content or satisfies any other suitablecriteria. Additionally or alternatively, in some embodiments, ininstances in which the content item corresponds to a submitted comment,block 204 can be omitted.

Additionally, note that, in some embodiments, process 200 can determinewhether the content creator is eligible to use the feature based on anysuitable combination of information. For example, in some embodiments,process 200 can determine whether the content creator is eligible to usethe feature based on a combination of a risk score associated with thecontent creator, information indicating a size of an audience of thecontent creator, information indicating previous violations of policiesby the content creator, and/or a manual review of content item(s). Insome embodiments, process 200 can combine the information in anysuitable manner. For example, in some embodiments, process 200 canassign a score to each item of information and can determine anaggregate score associated with the content creator based on the scorefor each item of information. In some embodiments, process 200 can thendetermine if the content creator is eligible to use the feature based ona determine of whether the aggregate score associated with the contentcreator exceeds a predetermined threshold.

If, at 204, process 200 determines that the content creator is noteligible to use the feature (“no” at 204), process 200 can proceed to206 and can reject the content item. In some embodiments, process 200can reject the content item in any suitable manner. For example, in someembodiments, process 200 can block upload of the content item to aserver associated with the media content platform. As another example,in some embodiments, process 200 can transmit a message to the contentcreator and/or cause a notification to be presented on a user deviceassociated with the content creator that indicates that the content itemhas been rejected. In some such embodiments, the message and/or thenotification can include any suitable information, such as a reason thecontent item has been rejected (e.g., due to previous violations by thecontent creator of one or more policies associated with the mediacontent platform, and/or any other suitable reason). As yet anotherexample, in some embodiments, in an instance in which the content itemcorresponds to a comment to be posted to a chat or a comments section,process 200 can inhibit the comment from being posted to the chat or thecomments section.

Note that, in some embodiments, in response to determining that thecontent creator is not eligible to use the feature, process 200 canallow the content item to be uploaded but can reject use of the featureby the content creator. For example, in an instance in which process 200determines that the content creator has not previously violated anypolicies associated with the media content platform but that the contentcreator is not eligible to use the feature due to other reasons (e.g.,because the content creator does not have a sufficiently large audience,because the content creator has not previously uploaded more than apredetermined number of content items, and/or due to any other suitablereason), process 200 can allow the content item to be uploaded to themedia content platform but can block or inhibit use of the feature bythe content creator.

If, at 204, process 200 determines that the content creator is eligibleto use the feature (“yes” at 204), process 200 can proceed to 208 andcan determine whether to selectively bypass review of the user-generatedcontent item. In some embodiments, process 200 can determine whether tobypass review of the content item based on any suitable information.

For example, in some embodiments, process 200 can determine whetherreview of the content item is to be bypassed based on a relative risk ofpublication of the content item if the content item includesobjectionable content. In some embodiments, a relative risk ofpublication of the content item can be determined based on any suitableinformation, such as an expected size of an audience of the content item(e.g., a predicted number of people likely to view the content item),whether the content creator will receive any revenue from viewers of thecontent item, an expected revenue the content creator will receive fromviewers of the content item, and/or any other suitable information. Notethat, in some embodiments, process 200 can determine that the contentitem is to be assigned a relatively high risk score in response todetermining that the content item satisfies any particular criteria. Forexample, in some embodiments, process 200 can determine that the contentitem is to be assigned a relatively high risk score in response todetermining that the content item is being offered in exchange forpayment by viewers (e.g., in instances in which the content item isexclusive content offered to users who pay for the content, in instancesin which the content item is a perk that is being offered to users whopay for a membership to a channel of content, and/or any other suitabletype of content).

In some embodiments, process 200 can determine a relative risk in anysuitable manner. For example, in some embodiments, process 200 canidentify multiple risk scores (e.g., a first risk score corresponding toan estimated audience size, a second risk score corresponding to anexpected revenue for the content creator from viewers of the contentitem, and/or any other suitable risk scores), and can combine themultiple risk scores in any suitable manner (e.g., as a weightedaverage, and/or in any other suitable manner). As a more particularexample, in an instance in which the content item corresponds to a videocontent item to be posted to a channel of content with a relativelylarge expected audience (e.g., based on the channel having more than apredetermined number of subscribers, based on the channel having morethan a predetermined number of previous views of content associated withthe channel, and/or based on any other suitable metric), process 200 candetermine that the relative risk of publication of the content item ishigh. As another more particular example, in an instance in which thecontent item corresponds to a video content item that has been monetizedin any suitable manner (e.g., due to insertion of advertisements,because the content item is being provided to users who have paid toreceive the content item, and/or in any other suitable manner), process200 can determine that the relative risk of publication of the contentitem is high. As yet another more particular example, in an instance inwhich the content item corresponds to a comment to be posted in acomments section, process 200 can determine that the relative risk ofpublication of the content item is low. In some embodiments, process 200can determine that content items associated with a relative risk ofpublication that exceeds a predetermined threshold are to be manuallyreviewed (that is, not bypassed for manual review, or “no” at 208). Notethat, in some embodiments, in an instance in which process 200determines that a content item is associated with a relative risk ofpublication that exceeds a particular predetermined threshold, process200 can determine that the content item is to be automatically rejected.

As another example, in some embodiments, process 200 can determine thatreview of the content item is to be bypassed in response to determiningthat the content item is similar to or matches previously uploadedcontent items that have previously been reviewed. As a more particularexample, in some embodiments, process 200 can determine that the contentitem is similar to or matches content items that have been previouslyreviewed and approved. As another more particular example, in someembodiments, process 200 can determine that the content item is similarto or matches content items that have been previously reviewed andrejected.

Note that, in some embodiments, process 200 can determine whether theuploaded content item is similar to or matches previously reviewedcontent items using any suitable technique or combination of techniques.For example, in some embodiments, process 200 can compare one or moreaudio and/or video fingerprints of the uploaded content item to adatabase of previously uploaded content items to identify matches topreviously uploaded and reviewed content items.

Additionally, note that, in some embodiments, process 200 can determinea degree of similarity required between the uploaded content item and atleast one previously reviewed content item to determine that review isto be bypassed. For example, in some embodiments, process 200 candetermine that the uploaded content item is to have at least apredetermined threshold of similarity to a previously reviewed contentitem (e.g., at least 70% similarity, at least 80% similarity, and/or anyother suitable threshold) to determine that review is to be bypassed. Insome embodiments, a predetermined threshold of similarity can bedetermined or set in any suitable manner. For example, in someembodiments, the predetermined threshold of similarity can be determinedby process 200 such that, when the predetermined threshold of similarityis applied, a predetermined range of content items are sent for manualreview. That is, in some embodiments, process 200 can adjust thepredetermined threshold of similarity over time based on a number ofcontent items that are uploaded to the media content platform, therebyallowing an amount of manual review that is performed to be adjusted.

Additionally, note that, in some embodiments, process 200 can determinewhether manual review is to be bypassed based on any suitablecombination of a relative risk of publication of the content item and asimilarity between the uploaded content item and previously uploadedcontent items. For example, in some embodiments, process 200 cancalculate a risk score indicating a relative risk of publication of thecontent item and a similarity score indicating a similarity between thecontent item and previously uploaded content items, and can determinewhether manual review is to be bypassed based on both the risk score andthe similarity score. As a more particular example, in some embodiments,process 200 can determine that manual review is not to be bypassed (“no”at 208) in response to determining that the risk score exceeds a firstpredetermined threshold and the similarity score is below a secondpredetermined threshold (that is, not sufficiently similar to apreviously uploaded content item). As another more particular example,in some embodiments, process 200 can determine that manual review is tobe bypassed (“yes” at 208) in response to determining that the riskscore is below a first predetermined threshold and the similarity scoreis above a second predetermined threshold.

In some embodiments, process 200 can determine that particular types ofcontent items are to be manually reviewed regardless of risk score. Forexample, in some embodiments, process 200 can determine that contentitems that are provided in exchange for payment by viewers of thecontent item (e.g., exclusive content that is provided to paying users,and/or any other suitable type of paid content) are always to bemanually reviewed. As another example, in some embodiments, process 200can determine that content items generated by content creators who areproviding paid content for a first time are to be manually reviewed.Note that, in some embodiments, paid content items can include videocontent, audio content, “super stickers,” “super comments,” (asdescribed above in connection with block 202) and/or any other suitabletype of content that is made available to viewers in exchange forpayment.

If, at 208, process 200 determines that review of the content item is tobe bypassed (“yes” at 208), process 200 can proceed to 210 and candetermine whether the content item is to be rejected as being abusivecontent. In some embodiments, process 200 can determine whether thecontent item is to be rejected based on whether it was determined atblock 208 that the uploaded content item is similar to or matches apreviously rejected content item. For example, in some embodiments,process 200 can determine that the uploaded content item is similar toor matches a previously rejected content item that was rejected due tothe content item including one or more types of objectionable content,and, in some such embodiments, process 200 can determine that contentitem is to be rejected. As another example, in some embodiments, process200 can determine that the uploaded content item is similar to ormatches a previously approved content item, and, in some suchembodiments, process 200 can determine that the content item is not tobe rejected. As yet another example, in some embodiments, process 200can determine a likelihood that the content item includes any particulartypes of objectionable content using any suitable technique(s). As amore particular example, in some embodiments, process 200 can use anysuitable classifier or combination of classifiers to determine alikelihood that the content item includes objectionable content. Notethat, in instances in which the content item corresponds to videocontent or audio content, process 200 can use a classifier that takes asinputs any suitable audio or video fingerprints of the content item. Insome embodiments, in instances in which the content item corresponds toa submitted comment, process 200 can use a classifier that takes as aninput any suitable text or links included in the comment to determine alikelihood that the text includes objectionable content or that the linkis to a site associated with objectionable content. Continuing with thisexample, in some embodiments, process 200 can determine that the contentitem is to be rejected in response to determining that a determinedlikelihood that the content item includes objectionable content exceedsa predetermined threshold (e.g., greater than 70%, greater than 80%,and/or any other suitable threshold).

If, at 210, process 200 determines that the content item is to berejected (“yes” at 210), process 200 can reject the content item at 206,as described above.

If, at 210, process 200 determines that the content item is not to berejected (“no” at 210), process 200 can proceed to 212 and can post thecontent item to the media content platform. In some embodiments, process200 can post the content item to the media content platform in anysuitable manner. For example, in some embodiments, process 200 canupload the content item to a particular channel associated with thecontent creator. As another example, in some embodiments, process 200can make a link to the content item available to other user devices tostream or download the content item. As another example, in someembodiments, process 200 can transmit a notification to the contentcreator indicating that the content item has been approved. In some suchembodiments, the content creator can then initiate posting of thecontent item in any suitable manner, for example, by selecting anysuitable selectable input that causes the content item to be madeavailable to other viewers. As yet another example, in an instance inwhich the content item corresponds to a live-stream of content, process200 can make the live-stream available to viewers after any suitabletime delay (e.g., after thirty seconds, after one minute, and/or afterany other suitable time delay). As still another example, in an instancein which the content item corresponds to a comment to be posted to acomments section or a chat, process 200 can cause the comment to beadded to a chat or a comments section in any suitable manner (e.g., atthe bottom of a chat, at a bottom of a comments section, in a pinnedposition of a chat or comments section, and/or in any other suitablemanner).

Referring back to block 208, if, at 208, process 200 determines thatreview of the content item is not to be bypassed (“no” at 208), process200 can proceed to 214 and can add the content item to a queue ofcontent items to be reviewed. As described above in connection withblock 104 of FIG. 1, in some embodiments, process 200 can add thecontent item to a queue of content items to be reviewed in any suitablemanner. For example, in some embodiments, the content item can be addedto the queue based on a date or a time the content item was received atblock 202 (e.g., such that items that were received earlier are placedearlier in the queue, and/or in any other suitable manner). As anotherexample, in some embodiments, the content item can be added to the queuebased on a popularity of other content items associated with the contentcreator. As a more particular example, in some embodiments, a contentitem associated with a content creator who has previously uploadedcontent items that have received more than a predetermined number ofviews, shares, comments, etc. can be placed higher in the queue. As yetanother example, in some embodiments, in an instance in whichpublication of the content item is time-sensitive (e.g., in an instancein which the content item is live-streamed content, in an instance inwhich the content item is a comment in a live chat, and/or in any othersuitable time-sensitive context), process 200 can add the content itemto a relatively high position within the queue (e.g., to a front of thequeue, and/or in any other suitable position), thereby allowing thecontent item to be manually reviewed more quickly.

As described above in connection with FIG. 1, in response to the contentitem being moved to the head of the queue of content items to bereviewer, the content item can be assigned to a reviewer. In some suchembodiments, as described above in connection with FIG. 1, the contentitem can be presented on a user device associated with the reviewer. Insome embodiments, the reviewer can then use the user device associatedwith the reviewer to enter any information associated with a decisionrelated to the content item, such as whether the content item isdetermined to include one or more types of objectionable content, framesor timestamps of the content item that include objectionable content ininstances where the content item corresponds to a video or to audiocontent, one or more policies violated by the content item, and/or anyother suitable information.

At 216, process 200 can receive a decision relating to the content item.In some embodiments, process 200 can receive the decision in anysuitable manner. For example, in some embodiments, a server executingprocess 200 can receive a message from a user device associated with areviewer who reviewed the content item. In some such embodiments, themessage can include any suitable information, such as an indication thatthe content item does not violate any policies associated with the mediacontent platform, an indication that the content item does violate oneor more policies associated with the media content platform, anindication of one or more types of objectionable content included in thecontent item, timestamps of the content item that include one or moretypes of objectionable content, and/or any other suitable information.

At 218, process 200 can determine whether the decision indicates apolicy violation.

If, at 218, process 200 determines that the decision does not indicate apolicy violation (“no” at 218), process 200 can post the content item tothe media content platform at 212, as described above.

If, at 218, process 200 determines that the decision does indicate apolicy violation (“yes” at 218), process 200 can proceed to 220 and cantransmit a message to the content creator indicating the policyviolation. In some embodiments, process 200 can transmit the message tothe content creator in any suitable manner. For example, in someembodiments, process 200 can transmit the message to an email addressassociated with the content creator. As another example, in someembodiments, process 200 can transmit the message in connection with auser account associated with the content creator and the media contentplatform. As a more particular example, in some embodiments, the messagecan be a notification that is presented on a home page of the useraccount associated with the content creator in connection with the mediacontent platform. This can, for example, provide precise feedback to thecontent creator as to policy violation and other reasons for taking anaction that inhibits the user-generated content from being madeavailable on the media content platform.

In some embodiments, the message can include any suitable information.For example, in some embodiments, the message can indicate the one ormore policies that the content item violates. As a more particularexample, in some embodiments, the message can indicate that the contentitem includes one or more types of objectionable content that violateone or more policies of the media content platform. As another example,in some embodiments, the message can indicate one or more portions ofthe content item that violate the policy of the media content platform,using any suitable timing information, such as timestamps of the contentitem, frames of the content item, and/or any other suitable timinginformation. As yet another example, in some embodiments, the messagecan indicate a penalty. As a more particular example, in someembodiments, the message can indicate that the content item has beenblocked from being uploaded to the media content platform. As anothermore particular example, in some embodiments, the message can indicatethat the content creator will be prohibited from uploading content itemsfor a predetermined duration of time (e.g., for a month, for 90 days,and/or for any other suitable duration of time). As yet another moreparticular example, in some embodiments, the message can indicate thatan account associated with the content creator will be terminated. Asstill another more particular example, in some embodiments, the messagecan indicate that the content creator will not be allowed to use thefeature for a predetermined duration of time (e.g., thirty days, ninetydays, and/or any other suitable duration of time). As still another moreparticular example, in some embodiments, the message can indicate thatthe content creator will not be allowed to stream live content for apredetermined duration of time (e.g., thirty days, ninety days, and/orany other suitable duration of time). As still another more particularexample, in an instance in which the content item corresponds to acomment to be posted to a chat or a comments section, the message canindicate that the user will not be allowed to post comments for aparticular duration of time to the comments section associated with therejected comment or to any comments section associated with the mediacontent platform.

Note that, in some embodiments, process 200 can determine the penaltybased on the policy violated by the content item. For example, in someembodiments, process 200 can determine the penalty based on a severityof the violation. As a more particular example, in some embodiments,process 200 can determine a first penalty (e.g., that the content itemwill be blocked) in response to determining that the policy violationcorresponds to inclusion of a first type of objectionable content in thecontent item and a second penalty (e.g., that the user accountassociated with the content creator will be terminated) in response todetermining that the policy violation corresponds to inclusion of asecond type of objectionable content in the content item. Additionally,note that, in some embodiments, process 200 can determine that aparticular penalty (e.g., account termination, prohibiting upload ofcontent for a predetermined duration of time, and/or any other suitablepenalty) is to be applied in response to determining that the contentcreator has violated one or more policies more than a predeterminednumber of times (e.g., more than three violations, more than fiveviolations, and/or any other suitable number). This can, for example,discourage policy violation with increasingly severe enforcementactions. Additionally, note that, in some embodiments, process 200 candetermine the penalty based on the type of content item. For example, insome embodiments, in instances in which the content item corresponds toupload of video content or audio content to be made available to otherviewers, process 200 can identify a penalty associated with a temporaryor permanent block of uploading of other video or audio content to themedia content platform. As another example, in some embodiments, ininstances in which the content item corresponds to a comment submittedto a chat or comments section, process 200 can identify a penaltyassociated with a temporary or permanent block of submission ofcomments.

Note that, in some embodiments, prior to proceeding to block 220,process 200 can add the content item to a second queue such that thecontent item is then reviewed for a second time by a second reviewer.That is, in some embodiments, in response to a determination that thedecision indicates a policy violation, process 200 can cause the contentitem to be reviewed for a second time by a second reviewer who isdifferent than the first reviewer to confirm that the content item doesviolate a policy of the media content platform.

At 222, process 200 can update an account associated with the contentcreator based on the policy violation. For example, in some embodiments,process 200 can update the account to indicate that the uploaded contentitem violated a particular policy. As another example, in someembodiments, process 200 can update the account to indicate that theuploaded content item was determined to include a particular type ofobjectionable content. As yet another example, in some embodiments,process 200 can update a tally that indicates a number of times thecontent creator has uploaded content items that have violated policiesassociated with the media content platform. Note that, in someembodiments, process 200 can use the information associated with theaccount of the content creator to determine a penalty for the contentcreator, as described above in connection with block 220.

At 224, process 200 can block the content item from upload to the mediacontent platform. Note that, in instances where the content item is alive-stream of content, process 200 can stop or inhibit the live-streamfrom being streamed to any user devices.

Note that, although processes 100 and 200 are described above asrelating to review of content items (e.g., videos, live-streamed videocontent, and/or any other suitable content item), in some embodiments,processes 100 and 200 can be used to review any other suitable type ofcontent, such as comments in a comment section, comments in a socialnetworking post, messages in a live chat, and/or any other suitablecomments. For example, in some embodiments, process 200 can determinewhether a particular comment is to be added to a queue of comments to bereviewed (e.g., at blocks 208 and 214) based on a classifier applied toa group of comments. As a more particular example, in some embodiments,process 200 can add a comment to a queue of comments to be manuallyreviewed in response to determining, using any suitable machine learningclassifier, that the comment is likely to contain one or more types ofobjectionable content (e.g., hate speech, objectionable language,objectionable images or links, and/or any other suitable type ofobjectionable content). Continuing with this example, in someembodiments, process 200 can add the comment to the queue of comments tobe reviewed (e.g., at block 214), and can proceed through the blocks ofprocess 200 to determine whether the comment is to be removed from thelive chat or comments section to which it was posted based on the manualreview. As another example, in some embodiments, processes 100 and 200can be used to review perks offered by a content creator in connectionwith purchase of a membership, as described above in connection withblock 202. As a more particular example, in some embodiments, process200 can determine whether a particular offered perk is to be added to aqueue of perks to be reviewed (e.g., at block 208 and 214) based on anysuitable information, such as whether complaints have been received byviewers who have purchased a membership, whether more than apredetermined threshold of viewers have canceled a purchased membership,and/or based on any other suitable criteria. Continuing with thisexample, process 200 can then proceed through the blocks of process 200to determine whether the offer of the perk is to be canceled ordiscontinued based on the manual review.

Turning to FIG. 3, an illustrative example 300 of hardware foridentifying abusive content items that can be used in accordance withsome embodiments of the disclosed subject matter is shown. Asillustrated, hardware 300 can include a server 302, a communicationnetwork 304, and/or one or more user devices 306, such as a contentcreator device 308 and/or a reviewer device 310.

Server 302 can be any suitable server(s) for storing information, data,media content, and/or any other suitable type of content. For example,in some embodiments, server 302 can store user-generated media contentuploaded by one or more content creator device 308. As a more particularexample, in some embodiments, the user-generated media content caninclude videos, movies, photos, slideshows, animations, documents,and/or any other suitable user-generated media content. As anotherexample, in some embodiments, server 302 can store information about oneor more content creators that have uploaded user-generated content toserver 302, such as whether a particular content creator has previouslyuploaded content that violates a particular policy of a media contentplatform, and/or any other suitable information. In some embodiments,server 302 can implement any suitable processes for queuing uploadedcontent items for review, storing a decision related to an uploadedcontent item (e.g., a decision indicating whether or not the contentitem contains objectionable content, and/or any other suitabledecision), and/or performing any other suitable functions. For example,in some embodiments, server 302 can implement any suitable blocks ofprocess 200, as shown in and described above in connection with FIG. 2.

Communication network 304 can be any suitable combination of one or morewired and/or wireless networks in some embodiments. For example,communication network 304 can include any one or more of the Internet,an intranet, a wide-area network (WAN), a local-area network (LAN), awireless network, a digital subscriber line (DSL) network, a frame relaynetwork, an asynchronous transfer mode (ATM) network, a virtual privatenetwork (VPN), and/or any other suitable communication network. Userdevices 306 can be connected by one or more communications links (e.g.,communications links 312) to communication network 304 that can belinked via one or more communications links (e.g., communications links314) to server 302. The communications links can be any communicationslinks suitable for communicating data among user devices 306 and server302 such as network links, dial-up links, wireless links, hard-wiredlinks, any other suitable communications links, or any suitablecombination of such links.

User devices 306 can include any one or more user devices. For example,in some embodiments, user devices 306 can include content creator device308. In some embodiments, content creator device 308 can be any userdevice suitable for capturing user-generated content, uploadinguser-generated content, receiving notifications indicating decisionsrelated to an uploaded content item, and/or for performing any othersuitable functions. As another example, in some embodiments, userdevices 306 can include reviewer device 310. In some embodiments,reviewer device 310 can be any user device suitable for viewing anuploaded content item, transmitting a decision related to the uploadedcontent item, and/or performing any other suitable function(s). In someembodiments, user devices 306 can include any suitable types of userdevices. For example, in some embodiments, user devices 306 can includea mobile phone, a wearable computer, a tablet computer, a desktopcomputer, a laptop computer, a vehicle entertainment system, a gameconsole, a television, and/or any other suitable user device.

Although server 302 is illustrated as one device, the functionsperformed by server 302 can be performed using any suitable number ofdevices in some embodiments. For example, in some embodiments, multipledevices can be used to implement the functions performed by server 302.

Although two user devices 308 and 310 are shown in FIG. 3 to avoidover-complicating the figure, any suitable number of user devices,and/or any suitable types of user devices, can be used in someembodiments.

Server 302 and user devices 306 can be implemented using any suitablehardware in some embodiments. For example, in some embodiments, devices302 and 306 can be implemented using any suitable general-purposecomputer or special-purpose computer. For example, a mobile phone may beimplemented using a special-purpose computer. Any such general-purposecomputer or special-purpose computer can include any suitable hardware.For example, as illustrated in example hardware 400 of FIG. 4, suchhardware can include hardware processor 402, memory and/or storage 404,an input device controller 406, an input device 408, display/audiodrivers 410, display and audio output circuitry 412, communicationinterface(s) 414, an antenna 416, and a bus 418.

Hardware processor 402 can include any suitable hardware processor, suchas a microprocessor, a micro-controller, digital signal processor(s),dedicated logic, and/or any other suitable circuitry for controlling thefunctioning of a general-purpose computer or a special-purpose computerin some embodiments. In some embodiments, hardware processor 402 can becontrolled by a server program stored in memory and/or storage of aserver, such as server 302. In some embodiments, hardware processor 402can be controlled by a computer program stored in memory and/or storage404 of user device 306.

Memory and/or storage 404 can be any suitable memory and/or storage forstoring programs, data, and/or any other suitable information in someembodiments. For example, memory and/or storage 404 can include randomaccess memory, read-only memory, flash memory, hard disk storage,optical media, and/or any other suitable memory.

Input device controller 406 can be any suitable circuitry forcontrolling and receiving input from one or more input devices 408 insome embodiments. For example, input device controller 406 can becircuitry for receiving input from a touchscreen, from a keyboard, fromone or more buttons, from a voice recognition circuit, from amicrophone, from a camera, from an optical sensor, from anaccelerometer, from a temperature sensor, from a near field sensor, froma pressure sensor, from an encoder, and/or any other type of inputdevice.

Display/audio drivers 410 can be any suitable circuitry for controllingand driving output to one or more display/audio output devices 412 insome embodiments. For example, display/audio drivers 410 can becircuitry for driving a touchscreen, a flat-panel display, a cathode raytube display, a projector, a speaker or speakers, and/or any othersuitable display and/or presentation devices.

Communication interface(s) 414 can be any suitable circuitry forinterfacing with one or more communication networks (e.g., computernetwork 304). For example, interface(s) 414 can include networkinterface card circuitry, wireless communication circuitry, and/or anyother suitable type of communication network circuitry.

Antenna 416 can be any suitable one or more antennas for wirelesslycommunicating with a communication network (e.g., communication network304) in some embodiments. In some embodiments, antenna 416 can beomitted.

Bus 418 can be any suitable mechanism for communicating between two ormore components 402, 404, 406, 410, and 414 in some embodiments.

Any other suitable components can be included in hardware 400 inaccordance with some embodiments.

In some embodiments, at least some of the above described blocks of theprocesses of FIGS. 1 and 2 can be executed or performed in any order orsequence not limited to the order and sequence shown in and described inconnection with the figures. Also, some of the above blocks of FIGS. 1and 2 can be executed or performed substantially simultaneously whereappropriate or in parallel to reduce latency and processing times.Additionally or alternatively, some of the above described blocks of theprocesses of FIGS. 1 and 2 can be omitted.

In some embodiments, any suitable computer readable media can be usedfor storing instructions for performing the functions and/or processesherein. For example, in some embodiments, computer readable media can betransitory or non-transitory. For example, non-transitory computerreadable media can include media such as non-transitory forms ofmagnetic media (such as hard disks, floppy disks, and/or any othersuitable magnetic media), non-transitory forms of optical media (such ascompact discs, digital video discs, Blu-ray discs, and/or any othersuitable optical media), non-transitory forms of semiconductor media(such as flash memory, electrically programmable read-only memory(EPROM), electrically erasable programmable read-only memory (EEPROM),and/or any other suitable semiconductor media), any suitable media thatis not fleeting or devoid of any semblance of permanence duringtransmission, and/or any suitable tangible media. As another example,transitory computer readable media can include signals on networks, inwires, conductors, optical fibers, circuits, any suitable media that isfleeting and devoid of any semblance of permanence during transmission,and/or any suitable intangible media.

Accordingly, methods, systems, and media for identifying abusive contentitems are provided.

Although the invention has been described and illustrated in theforegoing illustrative embodiments, it is understood that the presentdisclosure has been made only by way of example, and that numerouschanges in the details of implementation of the invention can be madewithout departing from the spirit and scope of the invention, which islimited only by the claims that follow. Features of the disclosedembodiments can be combined and rearranged in various ways.

What is claimed is:
 1. A method for identifying abusive content items, the method comprising: receiving, at a server from a user device associated with a user, a request to provide user-generated content on a media content platform; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criterion; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criterion, adding a review request to a queue of a reviewing user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision responsive to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device that indicates the at least one violated policy and a penalty to be administered to the user account corresponding to the user and (ii) a corresponding action is taken that inhibits the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device that indicates that the user-generated content is eligible for providing on the media content platform.
 2. The method of claim 1, wherein the user is a content creator and wherein the request to provide user-generated content includes an upload request to store the user-generated content on a storage device associated with the media content platform.
 3. The method of claim 1, wherein the user is a content creator and wherein the request to provide user-generated content includes an initiation request to start a livestream using the media content platform.
 4. The method of claim 1, wherein the user is a viewer and wherein the request to provide user-generated content includes a content feature that interacts with a content creator.
 5. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform by: calculating a risk score that indicates a risk associated with the user account of the user based on other user-generated content that has been previously provided to the media content platform; and comparing the risk score with a threshold risk score that corresponds to a content type of the of the user-generated content, wherein the user is deemed eligible to provide the user-generated content based on the comparison of the risk score with the threshold risk score.
 6. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform based on audience information associated with the user account of the user.
 7. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform based on location information associated with the user account of the user.
 8. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform based on enforcement actions for previous policy violations that are associated with the user account.
 9. The method of claim 1, further comprising determining whether the user is eligible to provide the user-generated content on the media content platform by transmitting a second review request to the queue of the reviewing user, wherein the second review request includes information associated with the user account corresponding to the user and prompts the reviewing user to determine whether the information associated with the user account violates at least one policy associated with the media content platform.
 10. The method of claim 1, further comprising, in response to receiving the request to provide the user-generated content on the media content platform, causing a timing indication to be presented that indicates when the decision responsive to the review request is to be provided.
 11. The method of claim 1, further comprising: receiving a flag from a viewer of the user-generated content on the media content platform, wherein the flag indicates a potential policy violation; and in response to receiving the subsequent review request, adding a subsequent review request to the queue of the reviewing user that previously reviewed the user-generated content, wherein the subsequent review request includes the user-generated content and the flag from the viewer and wherein the user-generated content is restricted from being provided on the media content platform until a decision responsive to the subsequent review request is received.
 12. The method of claim 1, wherein determining whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criterion further comprises: identifying second user-generated content that is similar to the user-generated content, wherein the second user-generated content has been previously reviewed; and determining that the user-generated content is not to be reviewed by the one or more reviewing users based on a similarity of the user-generated content to the second user-generated content.
 13. The method of claim 12, further comprising: determining that the second user-generated content was previously restricted from being made available on the media content platform; and in response to determining that the second user-generated content was previously restricted from being made available on the media content platform, performing the corresponding action that inhibits the user-generated content from being made available on the media content platform, wherein the first message indicates the restriction based on the second user-generated content.
 14. The method of claim 1, wherein determining whether the user-generated content is to be reviewed by the one or more reviewing users further comprises determining whether at least one of the user-generated content and content related to the user-generated content has been previously flagged by a threshold number of viewers.
 15. The method of claim 1, further comprising updating the user account associated with the user to indicate the at least one policy associated with the media content platform violated by the user-generated content.
 16. The method of claim 1, wherein the penalty to be administered to the user account corresponding to the user is based on a number of previous policy violations by the content creator.
 17. The method of claim 1, wherein the penalty to be administered to the user account corresponding to the user based on a severity of the at least one policy associated with the media content platform violated by the user-generated content.
 18. The method of claim 1, further comprising: receiving, from the user device, an appeal request of the decision, wherein the appeal request includes appeal text; adding the appeal request that includes the appeal text and the user-generated content to the queue of a second reviewing user that is different from the reviewing user that provided the decision indicating that the user-generated content is to be restricted from being provided on the media content platform; receiving, from a user device associated with the second reviewing user, an appeal decision corresponding to the appeal request; and determining at least one corresponding action to be performed on the user-generated content based on the decision and the appeal decision.
 19. The method of claim 1, wherein, in response to determining that the decision indicates that the user-generated content item does not violate the at least one policy associated with the media content platform, the second message indicates a time that the user-generated content is automatically made available on the media content platform.
 20. A system for identifying abusive content items, the system comprising: a memory; and a hardware processor that, when executing computer executable instructions stored in the memory, is configured to: receive, from a user device associated with a user, a request to provide user-generated content on a media content platform; in response to receiving the request, determine whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criterion; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criterion, add a review request to a queue of a reviewing user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determine whether a decision responsive to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device that indicates the at least one violated policy and a penalty to be administered to the user account corresponding to the user and (ii) a corresponding action is taken that inhibits the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device that indicates that the user-generated content is eligible for providing on the media content platform.
 21. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for identifying abusive content items, the method comprising: receiving, from a user device associated with a user, a request to provide user-generated content on a media content platform; in response to receiving the request, determining whether the user-generated content is to be reviewed by one or more reviewing users based on one or more bypass criterion; in response to determining that the user-generated content is to be reviewed based on the one or more bypass criterion, adding a review request to a queue of a reviewing user, wherein the review request includes the user-generated content and information associated with a user account corresponding to the user; and determining whether a decision responsive to the review request indicates that the user-generated content is to be provided on the media content platform, wherein: in response to determining that the decision indicates that the user-generated content item violates at least one policy associated with the media content platform, (i) a first message is transmitted to the user of the user device that indicates the at least one violated policy and a penalty to be administered to the user account corresponding to the user and (ii) a corresponding action is taken that inhibits the user-generated content from being provided on the media content platform; and in response to determining that the decision indicates that the user-generated content item does not violate the at least one policy associated with the media content platform, a second message is transmitted to the user of the user device that indicates that the user-generated content is eligible for providing on the media content platform. 